Papers with dynamic benchmark generator
Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts (2024.findings-emnlp)
Copied to clipboard
| Challenge: | evaluators of long-context vision language models (VLMs) have not kept up with the rapid development of open-weight long-constraint language models. |
| Approach: | They propose a dynamic benchmark generator for evaluating long-context reasoning in vision language models. |
| Outcome: | The proposed model can ignore irrelevant information when answering queries, showing that current models lack this capability. |